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Background: Nowadays, using new information technology (IT) has provided remarkable opportunities to decrease medical errors, 
support health care specialist, increase the efficiency and even the quality of patient's care and safety. 

Objectives: The purpose of this study was the identification of Hospital Information System (HIS) success and failure factors and the 
evaluation methods of these factors. This research emphasizes the need to a comprehensive evaluation of HISs which considers a wide 
range of success and failure factors in these systems. 

Materials and Methods: We searched for relevant English language studies based on keywords in title and abstract, using PubMed, Ovid 
Medline (by applying MeSH terms), Scopus, ScienceDirect and Embase (earliest entry to march 17, 2012). Studies which considered success 
models and success or failure factors, or studied the evaluation models of HISs and the related ones were chosen. Since the studies used 
in this systematic review were heterogeneous, the combination of extracted data was carried out by using narrative synthesis method. 
Results: We found 16 articles which required detailed analysis. Finally, the suggested framework includes 12 main factors (functional, 
organizational, behavioral, cultural, management, technical, strategy, economy, education, legal, ethical and political factors), 67 sub 
factors, and 33 suggested methods for the evaluation of these sub factors. 

Conclusions: The results of the present research indicates that the emphasis of the HIS evaluation moves from technical subjects to 
human and organizational subjects, and from objective to subjective issues. Therefore, this issue entails more familiarity with more 
qualitative evaluation methods. In most of the reviewed studies, the main focus has been laid on the necessity of using multi-method 
approaches and combining methods to obtain more comprehensive and useful results. 
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1. Background 

Nowadays, using new information technology (IT) has 
provided remarkable opportunities to decrease medical 
errors, support health care specialist, increase the effi- 
ciency and even the quality of patient's care and safety (1, 
2). On the other hand, there are numerous problems in 
the scope of IT-based systems in the field of health care; 
therefore, it causes a deep gap between the positive po- 
tential for IT to help health care organizations and their 
negative impacts. It means that a huge amount of money 
is invested in health information systems, but a signifi- 
cant portion of this money is wasted for inefficient sys- 
tems or not implemented ones (3). 

Evaluation means "the act of measuring or exploring 
properties of a health information system (in planning, 
development, implementation, or operation), the result 
of which informs a decision to be made concerning that 
system in a specific context" (4). An evaluation which is 



carried out based on suitable investment and approved 
techniques can cause the organization to have a forward 
movement (5, 6). Organizations require a comprehen- 
sive evaluation framework, which can help create and 
develop methods of information system evaluation (7); 
on the other hand, the identification of methods for in- 
formation systems evaluation can be possible through 
the identifying the success and failure factors of these 
systems (8). 

When we talk about success, we should identify what 
the criteria and parameters used for the evaluation of suc- 
cess are. Success is considered as a dynamic concept (9). 
Whether the system achieves its intended purpose from 
its establishment is what we mean by success; moreover, 
it should be carried out based on an anticipated time 
table and budget, while the project team and its users 
are satisfied with the results and this satisfaction should 



Implication for health policy/practice/research/medical education: 

The results of the present research indicate that the emphasis of HISs evaluation moves from technical subjects to human and organizational subjects 
and from objective to subjective issues. Therefore, this issue entails more familiarity with more qualitative evaluation methods. 
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be constant (10). Since the information system is compli- 
cated and multidimensional, it may succeed or fail in dif- 
ferent situations (11, 12). These days, due to increasing at- 
tention of different organizations to expenses related to 
the projects of information systems and gaining at least 
the minimum benefits from them, studying their success 
or failure has its special importance (13-15). There are vari- 
ous reports presented pertaining to the high rates of fail- 
ure in IT projects in industrial sector and health care or- 
ganizations especially in hospitals (3, 16-19). Kaplan in his 
study asserts that the rate of failure in implementation 
of IT in health care organizations of America is almost 50 
percent (20). 

Information systems are the combination of different 
elements, among which the measurement of some ele- 
ments is easier than that of the others. The evaluation 
framework which is simply concentrated on elements 
whose measurement is easier cannot introduce a per- 
fect framework in evaluation of information systems 
(21). Many case studies have been done, whose results 
are based on one or a few factors which affect the suc- 
cess of IT-based systems, so it mentions this important 
point that the findings of such literatures should be put 
together as a puzzle. In this case, a lot of benefits can be 
achieved from the significant studies done in this field 
by using different approaches (22). Just one comprehen- 
sive evaluation study can show whether a specific system 
can be successful in a special place or not. Moreover, each 
evaluation criterion must be measured through an ap- 
propriate method. A perfect evaluation should include 
all proper success factors (23, 24). 

Previous researchers have discussed the challenges 
of the evaluation of health information systems and 
the problems resulting from lack of unique conceptual 
framework to guide evaluation researches (25, 26). The 
most significant challenge in evaluation of information 
systems used in health care is that information systems 
can influence the improvement of treatment and the pa- 
tient's health level. Due to this potential influence on the 
patient's life, more accurate criteria should be applied for 
the evaluation of these systems (5, 11, 27). Of course, un- 
derstanding the effects, results and prerequisites neces- 
sary for successful implementation of information tech- 
nologies in health care require a multi-factor viewpoint 
(15). 

Some success and failure factors have either less or 
more importance in different information systems (28). 
The main focus of our study is placed on the HIS, a com- 
puter system is designed to support the hospital needs 
for comprehensive information including patient's infor- 
mation, clinical information and financial management 
(29, 30). The main purpose of this system is improvement 
the quality of care provided forpatients (31, 32 ).A1 though 
the integrated HIS leads to more efficient use of sources, 
it has not resulted in effective service offering, improve- 



ment quality of care and increased productivity in many 
countries (11, 33). Unfortunately, the measurement meth- 
odologies of different effects of these systems have not 
been improved along with the development of these sys- 
tems, while the future managers and users of HISs need 
accurate evaluation of these systems (11, 34). 

2. Objectives 

The purpose of this systematic review was the identifi- 
cation of the HIS success and failure factors and the eval- 
uation methods of these factors. This study emphasizes 
the need to a comprehensive evaluation of HISs which 
considers a wide range of success and failure factors in 
these systems. 

3. Materials and Methods 

We searched for relevant English language studies 
based on keywords in title and abstract, using PubMed, 
Ovid Medline (by applying MeSH terms), Scopus, Scien- 
ceDirect and Embase (earliest entry to March 17, 2012). In 
addition, we applied methods such as documentation 
review of relevant agencies (like Statistics and Informa- 
tion Technology Office in Ministry of Health and Medical 
Education) and databases that provide grey literature 
(like system for information on grey literature) for pub- 
lication bias control. Also, searching was supplemented 
by scanning bibliographies from identified studies. The 
key journals have been manually searched to find refer- 
ences which may not be found in databases and the list 
of references. Table 1 represents the used search strategy 
and Figure 1 shows a flowchart representing the search 
and way of choosing studies. 



Related article; identified through 
database searching (N = 475) 



Articles excluded because of their 
language (N - 6) 



Articles rejected by reviewing 
their titles (N = 382) 



The remaining and related articles 
assessed for more details (abstract) 
and eligibility (N = 87) 



Articles added after reviewing 
their bibliographies (N = 6) 



Articles rejected after their 
abstract review (N = 24) 



Screening the full text of 
remaining articles (N = 69) 



Full text articles excluded after 
complete review (N = 53) 



Articles finally reviewed and 
included (N = 16) 
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Figure 1. Flowchart of Search and Select the Included Articles in System- 
atic Review 



Table 1. Applied Search Strategy in Electronic Databases 
Database Search Strategy 



2. TITLE-ABSTR-KEY(success OR succeed OR fail 
OR failure OR evaluation OR assessment OR 
evaluate OR assess) 

3. 1 and 2 



Two reviewers independently examined all titles and 
abstracts. Cohen's Kappa coefficient was used to mea- 
sure the inter-rater agreement on the inclusion or exclu- 
sion of the articles (k=0.77). The difference between the 



ideas of the two reviewers was settled by asking the idea 
of the third reviewer. To control assessment bias, review- 
ers were blinded about each other's decisions and some 
information such as the names of journals and authors 
that could influence their decisions. Studies which con- 
sidered models and success or failure factors, or studied 
the evaluation models of the HIS and the related ones 
were chosen. The studies examining the information sys- 
tems in a specific area apart from the field of health care 
and in the field of public health as well as primary care in 
addition to non-English articles were excluded from the 
study. 

Assessing the quality of qualitative research has attract- 
ed much debate and there is little consensus regarding 
whether quality can or should be assessed in relation to 
qualitative research (35-37). Since most articles which en- 
tered in this study were review articles and data analysis 
was done on the words and phrases in the articles using 
meta synthesis, to avoid losing valuable insights that ex- 
isted in some studies that might be omitted in critical 
appraisal, we did not assess the quality of the selected 
studies. 

Since the studies used in this systematic review were 
heterogeneous, the combination of extracted data was 
carried out by using narrative synthesis method. Finally, 
the common and varying aspects of the factors and their 
evaluation methods used in the selected studies were 
determined. Then these factors and methods were classi- 
fied in different groups, and a framework was offered for 
the evaluation of success and failure factors in the HIS. 

4. Results 

Searching the online databases resulted in 475 articles. 
Initial screening of titles and abstracts rendered 63 ar- 
ticles eligible for further full-text review. Six additional 
articles were identified by reviewing the bibliographies, 
yielding a total of 69 articles. Based on reviewing the full- 
text of remaining articles, 53 articles were excluded since 
they did not correspond with the criteria and purpose of 
this study, and finallyl6 articles remained which required 
detailed analysis. The summaries of findings pertaining 
to these articles are presented in Table 2. Most selected 
studies considered factors such as management (n = 11), 
behavioural (n = 15), organizational (n =13) and economy 
(n = ll) as success factors (Table 3). Moreover, some evalu- 
ation methods such as interviews (n = 4), questionnaires 
(n = 5) and usability measurement methods (n = 7) were 
the most common methods in selected articles (Table 4). 



Ovid Medline 1. exp Hospital Information Systems/ 

2. exp Evaluation Studies as Topic/ 

3. Evaluation Studies.pt. 

4. (fail or failure or success or succeed).ti,ab. 

5. or/2-3 

6. 1 and 4 and 5 

7. limit 6 to English language 



Pub Med 



Embase 



Scopus 



1. "Hospital Information Systems"[Mesh] 

2. "Evaluation Studies as Topic"[Mesh] OR 
"Evaluation Studies" [Publication Type] 

3. fail[tiab] ORfailure[tiab] ORsuccess[tiab] 
ORsucceed[tiab] 

4. English[la] 

5. and/1-4 

1. exp Hospital Information Systems/ 

2. exp evaluation/ 

3. (fail or failure or success or succeed).ti, ab. 

4. and/l-3 

1. TlTLE("hospital information system") 

2. TITLE-ABS-KEY(success OR succeed OR fail 
OR failure OR evaluation OR assessment OR 
evaluate OR assess) 

3. 1 and 2 

4. SUB]AREA(mult OR medi OR nurs OR vete 
OR dent OR heal) 

5. 3 and 4 



Science Direct 



1. TITLE-ABSTR-KEY("hospital information 
system") 
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Table 3. The HIS Success Factors and Their Frequency in Selected Studies 


Success Factors 


Sub Factors 


Frequency (%) 


Reference No. 


Functional 


Preparation of the user requirements 


4(25) 


(38,43,44,28) 




Alignment of the role and design of the HIS (Task-technol- 

ncrv aHanHnn^ 

U iiV dk_l d LJ Ll vJ 1 1 J 


4(25) 


(3,28,39,41) 




Pi pyi hi 1 i t"\7 triWTA rH <; H\7n^ m ir r ri ;i n crpc anrl rri^n (jpc i n t~ri p 

1 1CA.1L/ 111 l J LU Well LI J LI V llclllllL \_lltlllgC J all LI LllCllltiLj 111 LI 1L 

organizational context 




f?K 41 4^1 44"\ 




Added functionality are provided by the HIS, enabling us- 
ers to provide new or better services 


3 (18.75) 


(28, 41, 44) 




Improve clinical performance and outcomes 


4(25) 


(15,41, 44,45) 




In general 


9(56.25) 


(3, 15, 17, 28, 38, 39, 41, 43, 44) 


Organizational 


Collaboration and cooperation 


6 (37.5) 


(17,28, 38,41, 42, 44) 




Participation in decision-making 


3 (18.75) 


(11, 41, 44) 




Work from the workflow 


3 (18.75) 


(?R 44 4fi1 




^unnnrt frnm FiicrFipr lpvpl nrtrani73t"inn*; 

JUUUUl L 1 1 KJ 1 11 llli^HLl 1CVC1 UlgalllZ.aLlUllJ 




(28 41 44) 




M;ikp imnlpmpntatinn a tr^nsn^rpnt nrorpss within thp 

IVlulVC IIIIUICUICIIIIILIUII Cl LldlljUAlCllL U1ULL JJ VV1L11111 L11C 

organization 


2 (12.5) 


(28, 44) 




Organizational stability 


3 (18.75) 


(38, 44,46) 




Rate of hospital independence and authority 


1(6.25) 


(41) 




Organizational capacity for changes 


2 (12.5) 


(39,44) 




In general 


13 (81.25) 


(3, 11, 15, 17, 23, 28, 38, 39, 41-44, 
46) 


Behavioral 


User involvement 


4(25) 


(28, 42,44,46) 




User engagement and commitment 


6 (37.5) 


(28, 39-42,46) 




Resistance to changes 


3 (18.75) 


(28, 38,41) 




User knowledge and skills 


6(37.5) 


(3,38,40-43) 




Stakeholder, user and patient satisfaction 


15(93.75) 


(3,11,15,17, 23,28, 38-46) 




Motivational activities 


5(31.25) 


(17,29,40,41,46) 




User acceptance (perceived system ease of use, perceived 
system usefulness) 


7(43.75) 


(29,38-43) 




In general 


15(93.75) 


(3,11,15,17, 23,29,38-46) 


Cultural 


Understand health care as a specific culture 


1(6.25) 


(29) 




Understand the local culture (such as attention to cultural 
differences between public and private hospitals as well as 

Hpvplonin ct a nH HpvplonpH ronntripO 

UUVClUUUltl ull\J UCVCHJIJCU V.UL111 LI ICj J 


3 (18.75) 


(3,29,44) 




PrpnarpHnpcs 3nH willincrnp*;*; t"n\A/^rrl<; nilt"iir^l rli^ncrp 

1 ICUHICUUCjj clllll Will 11 lilllL j j LUWdlUj LllllUl ell LllclllwL 

(professional culture) 








Expectations of users 


3 (18.75) 


(29,38,41) 




In general 


6 (37.5) 


(3,17,29,38,41, 44) 


Management 


Managers commitment 


5(31.25) 


(29,38,40,41, 44) 




Formulation and expression of a clear vision for the enter- 
prise showing the HIS as part of it 


1(6.25) 


(44) 




Setting clear goals and instructions 


3 (18.75) 


(3,17,44) 




Flexible planning 


3 (18.75) 


(29,41, 44) 




Prospective and proactive control 


1(6.25) 


(29) 




Coping with the impact of change 


3 (18.75) 


(3,28,44) 




Internal communication and clear feedback 


3 (18.75) 


(28, 41, 44) 




Having a strategy 


4(25) 


(28, 40,41,44) 
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Handling the diversity within stakeholder goals 


2(12.5) 


(28,38) 




Using formal project management methodology 


2 (12.5) 


(38,46) 




Dedicate, availability and prioritize of competitive hospital 
resources (human, financial and physical resources and 
time) 


6 (37.5) 


(3, 17, 38, 41, 43, 44) 




Identify and mitigate risk (risk management) 


1(6.25) 


(17) 




Consider IT implementation as a change process 


1(6.25) 


(28) 




Understanding socio-technical nature of HISs 


3(18.75) 


(11, 15, 17) 




Regular evaluations and using their results at different 
stages of HIS life cycle 


2 (12.5) 


(40,44) 




In general 


11(68.75) 


(3, 11, 15, 17, 28, 38, 40, 41, 43, 
44, 46) 


lech mi al (system 
quality, information 
quality and service 
quality) 


Integration with Legacy system 


3(18.75) 


(28, 40, 44) 




Interoperability and Interconnectivity 


2(12.5) 


(17, 42) 




Usability 


7(43.75) 


(28,38,40-42, 44,46) 




Balance between flexibility and stability of IT 


2 (12.5) 


(28,40) 




Reliable technical infrastructure or network, 


2(12.5) 


(3, 38) 




Complexity of the system 


3(18.75) 


(17,28,38) 




Information quality (relevancy, usefulness, completeness, 
etc.) 


8(50) 


(3, 11, 15, 23, 28, 41, 42, 46) 




Response time (system speed) 


5 (31.25) 


(28, 40-42, 44) 




System security 


3(18.75) 


(40-42) 




Service quality (the support provided by the information 
department, the support provided by the maintenance 
company) 


5 (31.25) 


(28,40-43) 




Quality of user documentation 


2(12.5) 


(40,42) 




Flexibility and adoptability, enabling future functional and 
technical changes 


5 (31.25) 


(28,38, 41, 42, 44) 




Using proper standards, coding and nomenclature 


1(6.25) 


(40) 




In general 


13(81.25) 


(3, 11, 15, 17, 23, 28, 38, 40-44, 
46) 


Strategy 


National, regional, organizational 


2 (12.5) 


(28,41) 




Accepted also at lower levels 


1(6.25) 


(28) 




Alignment between system strategies and hospital strate- 
gies 


3 (18.75) 


(3, 38,46) 




In general 


7(43.75) 


(3,28,38,40,41, 44,46) 


Economy 


Return on investment (material or immaterial) 


4(25) 


(15, 28, 40, 41) 




Justification of increase of costs 


4(25) 


(15,28,41, 45) 




Sufficient funding 


4(25) 


(28,40,41, 44) 




In general 


11(68.75) 


(3, 11, 15, 17, 28, 40, 41, 43-46) 


Education 


Sufficient training to make the best out of the daily opera- 
tion 


3 (18.75) 


(17, 28,40) 




Sufficient training to provide an understanding of its limi- 
tations and future potentials 


1(6.25) 


<n> 




In general 


7(43.75) 


(17, 28,40-44) 


Legal 


Compliance with legal requirements 


1(6.25) 


(28) 




Know what the legal constraints/opportunities are 


1(6.25) 


(28) 
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In general 


2 (12.5) 


(28, 38) 




Ethical Compliance with existing ethical rules in affairs manage- 


1(6.25) 


(28) 




ment 








Privacy and confidentiality 


1(6.25) 


(38) 




In general 


3(18.75) 


(28, 38, 41) 




Political Political games/conflicts 


2(12.5) 


(28, 38) 




Willingness towards investment on IT systems 


1(6.25) 


(28) 




Reliable external partners 


2 (12.5) 


(28, 38) 




In general 


5(31.25) 


(3,28,38,41, 44) 




Table 4. The HIS Evaluation Methods and Their Frequency in Selected Studies 


Evaluation Methods 




Frequency (%) 


Reference No. 


Analysis of work procedure 




1(6.25) 


(47) 


Stakeholder analysis 




1(6.25) 


(47) 


Organizational readiness 




1(6.25) 


(44) 


Framework for assessment of strategies 




1(6.25) 


(47) 


Thinking-aloud 




2(12.5) 


(15, 45) 


Cognitive assessment 




2(12.5) 


(15, 47) 


Cognitive walkthrough 




2(12.5) 


(15, 47) 


Heuristic evaluation 




2(12.5) 


(15, 47) 


Balanced scorecard 




1(6.25) 


(47) 


Risk assessment 




3 (18.75) 


(3, 40,47) 


Focus group interview 




2(12.5) 


(40,47) 


Delphi 




1(6.25) 


(47) 


Social network analysis 




2(12.5) 


(15, 47) 


Randomized controlled studies 




3 (18.75) 


(15, 45, 47) 


Use proxy outcomes (decreased medication errors, improved adherence to practice 
guidelines, and improved quality of documentation) 


2 (12.5) 


(45, 47) 


Interviews 




4(25) 


(23,39,45,47) 


Questionnaires 




5(31.25) 


(11, 23,39,45,47) 


Functionality assessment 




1(6.25) 


(47) 


Design-reality gap model 




1(6.25) 


(3) 


Logical framework approach (LFA) 




1(6.25) 


(47) 


Work sampling 




2(12.5) 


(23,39) 


SWOT 




1(6.25) 


(47) 


Studying the existing documents and Chart review 




2(12.5) 


(23,39) 


Technical verification 




2 (12.5) 


(11, 47) 


Video recording 




3 (18.75) 


(11, 45, 47) 


Time studies 




2(12.5) 


(23,45) 


Kept the log 




2(12.5) 


(11, 23) 


Effect assessment 




1(6.25) 


(47) 


Field study 




1(6.25) 


(47) 


Cost minimization analysis 




1(6.25) 


(11) 


Cost effectiveness analysis 




3 (18.75) 


(11, 40, 45) 


Cost utility analysis 




3 (18.75) 


(11, 40,45) 
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Cost benefit analysis 



5. Discussion 

The results of the present research indicate that the em- 
phasis of HISs evaluation moves from technical subjects 
to human and organizational subjects and from objec- 
tive to subjective issues. Therefore, this issue entails more 
familiarity with more qualitative evaluation methods. 

It has been proved that the application of HISs can create 
basic changes in culture, policy and authority, which link 
professional groups to one another in an organization. 
However, these basic subjects have not been identified in 
many of success models. These issues result in the inabil- 
ity of these models to interpret some cases of failure (39). 
Many studies published about the success of information 
systems have been carried out by focusing on the model of 
Delone and McLean (48, 49). Since in this model some im- 
portant factors have not been considered, the mentioned 
studies cannot present a comprehensive model in this 
field. For instance, they deal with supporting the top man- 
agers and the involvement of users, and point out that 
these factors can influence the level of success, but these 
variables are not considered in this model. Moreover, other 
factors like culture and organizational characteristics are 
not considered in this model. The changes of procedures 
and culture are among the obstacles which are reported 
in widespread use of health care information systems (41). 
Other studies emphasized on organizational and manage- 
ment factors such as management commitment, a cham- 
pion and his/her characteristics (28, 38, 41, 51, 52). 

The findings indicate that, among different factors of 
success, user satisfaction to measure HISs success is of 
special significance (3, 11, 15, 17, 23, 28, 38-46). This factor 
is probably the most widespread factor used in measure- 
ment of success (11, 41, 42). Even though some research- 
ers may doubt about the reliability and appropriateness 
of using the user satisfaction to measure the success of 
information systems (52, 53), the researches which have 
been carried out, have proved that satisfaction is a use- 
ful criterion to measure the success of the system (11, 32, 
54, 55). However, in the field of health care information 
systems, only four percent of researches have used the 
criterion of the influence of user satisfaction, whereas 
this rate reaches almost 20 percent in the researches 
of information systems (56). The findings show that, in 
published literatures, special attention has been paid to 
the economy factors and their evaluation (3, 11, 15, 17, 28, 
38, 40, 41, 43-46). On the other hand, when the efficiency 
of investment in IT is studied, some problems manifest 
themselves as follows (11): 

1- The expenditures are mostly indirect, so their calcula- 
tion is difficult 

2- The benefits and organizational impacts are often in- 
tangible, so their realization may take a long time 



3(18.75) (11,40,45) 



In spite of these problems, findings show that there 
are appropriate and verified methods for evaluating 
economy aspects which provide useful information in 
this area. The other point which seems noticeable about 
the success factors of the HIS refers to two of selected ar- 
ticles indicating that the evaluation of HISs is one of the 
success factors (40, 44). We believe that the quality and 
quantity of evaluations and using their results at differ- 
ent stages of the HIS life cycle can be an influential and 
significant factor in the success of such systems, and this 
factor can be studied through methods like interviews 
and studying the existing documents. 

The findings reveal that, among the suggested methods 
for the evaluation of success or failure factors of HISs, us- 
ing questionnaires (11, 23, 39, 45, 47) and interviews (23, 
39, 45, 47) are more emphasized. Questionnaires are the 
best approved method for the evaluation of personal 
opinion, perception and attitude which is widely used 
in information systems and health researches. A compre- 
hensive interview and focus group interview are effective 
on understanding how and why events have happened 
or will occur and on the perception of users about how 
affairs can be done more efficiently (39). 

Among the evaluation methods extracted for this re- 
search, some methods are based on the retrospective na- 
ture (like functionality assessment). The other kinds such 
as balanced scorecard and delphi methods might present 
a guideline for planning and revision HISs, which provide 
formative evaluation (57). Some methods may not meet 
the specific information needs in the related field com- 
pletely, but can be used as valuable supportive means 
in the field of evaluation. As the symptoms of a disease 
comprise a part of the pattern of that disease all these 
methods present a pattern to show the success or failure 
(8). The presented methods are adapted from different 
sciences including psychology, social science, computer 
and health informatics sciences. Applying adapted evalu- 
ation methods requires methodological skills, discipline, 
innovation and flexibility to adapt the chosen method 
with the intended case, its situation and specifically in- 
formation needs (8, 24). 

The major limitations of this study are that some select- 
ed studies deal with only success and failure factors (28, 
38, 42, 43, 46), or focused on evaluation methods (47), but, 
due to the significance of these studies, they are consid- 
ered and reviewed in this research. In addition, among 
the extracted factors, there were some factors which 
could be placed in different groups and several discus- 
sion sessions took place to choose the most appropriate 
category for each factor. Another limitation is that only 
the articles written in English were selected; therefore, 
there might be some noteworthy articles in this field 
published in other languages. 
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In most of the reviewed articles, the main focus has been 
laid on the necessity of using multi-method approaches 
and combining methods to obtain more comprehensive 
and useful results. The integrating of different methods 
can be beneficial to find an inclusive answer to evaluation 
questions. The integrity of supplementary methods, data 
sources, theories and observers are studied under the 
term of triangulation (58). The combination of qualita- 
tive data gathering approaches (such as observations and 
interviews) and quantitative data gathering approaches 
(such as questionnaires and work sampling) provides a 
good opportunity through triangulation to improve the 
quality of results (59). 



Finally, in this research, the extraction of key concepts 
of each study was carried out through applying meta 
synthesis, and such concepts were put together and clas- 
sified, then the suggested framework was formed to eval- 
uate the HIS success [ Table 5 ]. This framework includes 
12 main factors, 67 sub factors, and 33 suggested methods 
for the evaluation of these sub factors. One can deter- 
mine the appropriate method based on this information 
and with a general review of the listed methods and fac- 
tors, or he/she can create better and more comprehensive 
methods based on his/her information needs. Of course, 
more attempts and investigations are still necessary to be 
done in this field. 



Table 5. The HIS Success Factors and Their Suggested Evaluation Methods 


Success Factors 


Sub Factors 


Evaluation Methods 


Functional 


Preparation of the user requirements 


Analysis of work procedure, Stakeholder analysis, 
Organizational readiness, Framework for assessment of 
strategies 




Alignment of the role and design of the HIS (Task- 
technology adaption) 


Thinking-aloud, Cognitive assessment, Cognitive 
walkthrough, Heuristic evaluation, Analysis of work 
procedure 




Flexibility towards dynamic changes and changes in 
the organizational context 


Organizational readiness, Balanced scorecard, Risk as- 
sessment 




Added functionality are provided by the HIS, enabling 

1 1 core t~n n Hrio n 01 a/ notfo rco n riroc 
USC1S LU piuviuc 11CW Ul Ucllcl 


Focus group interview, Delphi, Social network analysis, 

jLdKcllUlUcl dlldiyMb 




Improve clinical performance and outcomes 


Randomized controlled studies, Use proxy outcomes 

( f\f*rrf*a<zf*r\ mpHir3t"inn prrnrc imnrnvprl z\c\ hprpnrp tri 

l LIC\_1 CcIjCVJ 1 11L LI 1L. ill IkJ 1 1 C11U1 J, 1111.IJ1 U VCVJ CH-IIICICIILC LU 

practice guidelines, and improved quality of documen- 
tation) 


Organizational 


Collaboration and cooperation 


Social network analysis, Stakeholder analysis 




Participation in decision-making 


Focus group interview 




Work from the workflow 


Social network analysis 




Support from higher level organizations 


Stakeholder analysis, Interviews (non-standardized) 




Make implementation a transparent process within 
the organization 


Focus group interview, Social network analysis, Stake- 
holder analysis 




Organizational stability 


Social network analysis, Organizational readiness 




Rate of hospital independence and authority 


Interviews 




Organizational capacity for changes 


Organizational readiness 


Behavioral 


User involvement 


Social network analysis, Stakeholder analysis, Focus 
group interview, Questionnaires 




User engagement and commitment 


Focus group interview, Social network analysis, Stake- 
holder analysis 




Resistance to changes 


Organizational readiness 




User knowledge and skills 


Focus group interview, Questionnaires 




Stakeholder, user and patient satisfaction 


Focus group interview, Questionnaires 




Motivational activities 


Focus group interview, Personal interviews 




User acceptance (perceived system ease of use, per- 
ceived system usefulness) 


Focus group interview, Questionnaires 



Cultural Understand health care as a specific culture Analysis of work procedure 
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Understand the local culture (such as attention to cul- 
tural differences between public and private hospitals 
as well as developing and developed countries) 


Analysis of work procedure, Functionality assessment, 
Design — reality gap model 




Preparedness and willingness towards cultural change 
(professional culture) 


Organizational readiness 




Expectations of users 


Focus group interview 


Management 


Managers commitment 


Logical framework approach (LFA) 




Formulation and expression of a clear vision for the 
enterprise showing the HIS as part of it 


Balanced scorecard, Framework for assessment of strate- 
gies 




Setting clear goals and instructions 


Balanced scorecard, Framework for assessment of strate- 
gies 




Flexible planning 


Documents and chart review 




Prospective and proactive control 


Logical framework approach (LFA) 




Coping with the impact of change 


Organizational readiness, Work sampling 




Internal communication and clear feedback 


Social network analysis 




Having a strategy 


Framework for assessment of strategies 




Handling the diversity within stakeholder goals 


Stakeholder analysis, Organizational readiness 




Using formal project management methodology 


Analysis of work procedure, Balanced scorecard, Frame- 
work for assessment of strategies 




Dedicate, availability and prioritize of competitive 
hospital resources (human, financial and physical 
resources and time) 


Risk assessment, SWOT assessment 




Identify and mitigate risk (risk management) 


Risk assessment, Design— reality gap model 




Consider IT implementation as a change process 


Analysis of work procedure, Framework for assessment 
of strategies 




Understanding socio-technical nature of HISs 


Interviews, Questionnaires, Delphi 




Regular evaluations and using their results at different 
stages of HIS life cycle 


Interviews, Studying the existing documents and Chart 
review 


Technical 
(system qual- 
ity, information 
quality and 
service quality) 


Integration with Legacy system 


Technical verification 




Interoperability and Interconnectivity 


Technical verification 




Usability 


Cognitive assessment, Cognitive walkthrough, Heuristic 
evaluation, Video recording, Thinking-aloud, Work 
sampling, Time studies, Kept the log 




Balance between flexibility and stability of IT 


Organizational readiness against change 




Reliable technical infrastructure or network, 


Technical verification 




Complexity of the system 


Technical verification 




Information quality (relevancy, usefulness, complete- 
ness, etc.) 


Chart review, Questionnaires 




Response time (system speed) 


Questionnaires, Time and motion studies, Work sam- 




System security 


Technical verification 




Service quality (the support provided by the informa- 
tion department, the support provided by the mainte- 
nance company) 


Review of contracted task agreement and chart review, 
Questionnaires, Interviews 




Quality of user documentation 


Thinking-aloud, Questionnaires, Interviews 




Flexibility and adoptability, enabling future functional 
and technical changes 


Technical verification 




Using proper standards, coding and nomenclature 


Technical verification 
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Strategy 



National, regional, organizational 



Framework for assessment of strategies 





Accepted also at lower levels 


Focus group interview 




Alignment between system strategies and hospital 
strategies 


Framework for assessment of strategies 


Economy 


Return on investment (material or immaterial) 


Delphi, Effect assessment, Field study, Cost minimiza- 
tion analysis, Cost effectiveness analysis, Cost utility 
analysis, Cost benefit analysis 




Justification of increase of costs 


Delphi, Cost effectiveness analysis, Cost utility analysis, 
Cost benefit analysis 




Sufficient funding 


Delphi, Cost effectiveness analysis, Cost utility analysis, 
Cost benefit analysis 


Education 


Sufficient training to make the best out of the daily 
operation 


Functionality assessment, Work sampling, Time studies 




Sufficient training to provide an understanding of its 
limitations and future potentials 


Focus group interview 


Legal 


Compliance with legal requirements 


Field study, Review of legal documents 




Know what the legal constraints/opportunities are 


SWOT, Interviews 


Ethical 


Compliance with existing ethical rules in affairs man- 
agement 


Focus group interview 




Privacy and confidentiality 


Focus group interview, Chart review 


Political 


Political games/conflicts 


SWOT, Delphi 




Willingness towards investment on IT systems 


Interviews, Questionnaires 




Reliable external partners 


Chart review and review of contracts as well as history of 
partners activities 
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